1. Technical Field
This invention relates in general to methods for processing digital images. One embodiment is a method of acquiring a digital image of large dynamic range using a sensor of relatively lower dynamic range capability.
2. Description of the Related Art
Image sensors are used in many types of applications of image acquisition. The two main technologies of image sensors are the so-called CCD (Charge Coupled Devices) and the x-y addressable CMOS devices.
These devices are basically composed of a set or array of photo-detectors that convert the incident light in an electric signal representative of the amount of light impinging on the pixel. The array may be read for producing an image based on the detection pattern of the impinging light. The integration time may be controlled through commonly implemented techniques and usually it is the same for all the photo-detectors of the array.
The dynamic range (briefly DR) of a device for image acquisition is the separation between the maximum light intensity that the elementary photo-detectors of the sensor may discriminate without saturating and the minimum light intensity that they are able to sense.
Image sensors, that integrate the electric current generated by impinging photons such as the CMOS devices, have a dynamic range that is limited by the amount of charge that may be discriminately accumulated in correspondence of the image pixels.
Moreover, using a linear digital image acquisition system, the allowed dynamic range is a compromise between the number of bits per pixel and the integration time, as schematically depicted in FIG. 1.
In order to capture the faintest pixels, the exposure time t1 (combination of integration time and gains) may be set to a value that would saturate pixels at a light intensity for example of about ⅓*Imax, wherein Imax is the maximum intensity that can be detected by the sensitive elements of the array before reaching saturation.
The quantization step of the sensed light intensities will be given by the ratio between the intensity ⅓*Imax and the number of gray levels to be discriminated.
Should ⅔*Imax be the intensity that saturates the sensitive elements, for the same number of gray levels the quantization step will be larger (twice the value for the previous case).
The dynamic range changes by varying the exposure time. For a given number of bits per pixel, larger dynamic ranges imply larger quantization errors and increasingly sensitive noise effects in the histogram of intensities relative to darker portions.
Many techniques for extending the dynamic range of linear sensors [1-5] have been developed.
Nayar WO 00/79784 proposed a technique for extending the dynamic range of a linear sensor having a relatively low dynamic range by employing an optical mask over the sensor defining a fixed spatial attenuation pattern, or an array of cells with a controllable spatial attenuation, as shown in FIG. 2.
The acquired image is normalized in terms of the exposure after a calibration phase that corrects the response of the image sensor in combination with the mask and thus is nonlinear, as schematically illustrated in FIG. 3. The normalized image is then interpolated for recovering saturated or black pixels.
A largely used technique for generating images substantially without saturated or black pixels is schematically depicted in FIG. 4 and is disclosed in High Dynamic Range Imaging, Acquisition, Display, and Image-Based Lighting, authored by Erik Reinhard, Greg Ward, Sumanta Pattanaik, Paul E. Debevec, 29 Nov. 2005, Morgan Kaufmann Publishers Inc.
According to this known technique using a sensor of intrinsically low dynamic range, a plurality of shots of the same scene are taken with different integration times. An image with a high dynamic range is then obtained by combining the pixels of the pictures taken (four in the illustration of FIG. 4). Finally, the intensities of the pixels in the relatively low dynamic range of the sensor are scaled by using a certain compression function.
A problem with this technique is that several shots (pictures) must be taken without moving the photo-camera and the depicted objects/subjects must be still. Under these conditions all the pictures represent exactly the same scene, otherwise it would be very difficult to align them correctly.
The published application WO 01/63914 discloses a method for acquiring an image of large dynamical range using an image sensor with a relatively low dynamical range exposed to the incident light coming from the scene to be acquired. The image sensor has a plurality of photo-sensitive elements (photodetectors) arranged in a two-dimensional array and each sensitive element has a sensitivity level that is fixed according to a pre-established spatial distribution.
In practice, the image sensor has a pre-established spatially varying sensitivity pattern according to which the sensitivity level of each light-sensing element is established. The sensitivity of each light-sensing element is permanently fixed through masked etching steps when the image sensor is fabricated.
The main difference in respect to the solution of FIG. 2 consists in that there is not any optical mask interposed between the image sensor and the scene to be acquired, but the image sensor is composed of photo-sensitive elements of different sensitivity.
The published application WO 2005/024948 discloses a method for enhancing the dynamic range of a linear sensor of relatively low dynamic range by acquiring image pixel values with one of two different integration times, according to a pre-established spatial pattern, for example of the type depicted in FIG. 5.
The structure of the sensor described in the above identified publication has two buses or equivalent circuit means for controlling the pixel integration time according to a certain spatial pattern. With this technique luminous pixels of the image may be acquired with a shorter integration time and dark pixels with a longer integration time. It is thus possible to reconstruct an image of the scene that has a reduced number of saturated or dark pixels than the image that could be obtained with a single uniform integration time for all the pixels.
A drawback of this method consists in that the definition of the spatial pattern of the two different integration times heavily influences the quality of the reconstructed images. In particular, the integration time differences set for an image with a strong contrast may not be appropriate for acquiring an image of a significantly lighter contrast. Moreover, an erroneous choice of the spatial exposition pattern could make luminous zones of the image be acquired with a too long integration time and vice versa for the dark zones.
The U.S. Pat. No. 6,078,037 discloses an apparatus and a method for obtaining enhanced digital images. A light sensitive element such as a photo-diode is employed to sense a light level and a plurality of identical storage elements are associated with the sensitive element through configuration switches for charging the storage elements. The storage elements are connected in sequence to the light sensitive element by driving the switches with appropriate control phases. As an alternative, the control phases of the configuration switches may overlap partially to each other.
Drawbacks of this prior apparatus consist in that a dedicated logic circuitry may be needed for establishing the integration time of each storage element, and the intensity values are not taken simultaneously. Moreover, the charge characteristic of a certain storage element is modified at a certain instant and another storage element is connected electrically in parallel thereto.